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Executive Summary 

Most National Library of Medicine (NLM) customers have access to the Internet, based 
on the results of this survey of NLM online database users, although the majority of these Inter- 
net-capable users do not currently access NLM databases via Internet. 1 The survey results pro- 
vide a strong basis for the transition to Internet-based delivery of NLM online database services, 
including Internet Grateful Med" and the NLM family of World Wide Web sites. '" 

Internet access is uneven, however, according to the survey results, especially in rural ar- 
eas and at hospitals. About 36 percent of rural users and almost half of users at hospitals do not 
have Internet access. Likewise, about half of users at rural libraries do not have Internet access. 
These results reinforce the need for continuing special outreach efforts directed at improving ac- 
cess for rural and hospital-based users and rural libraries. 

Other survey highlights: 

• Of the 75 percent of online users with Internet access, about three-quarters use Internet to 
access the World Wide Web. But only one-quarter use Internet to access NLM databases, 
suggesting a significant untapped potential for providing NLM databases via the Internet. 

• Of the 25 percent of users without Internet access, almost three-fifths intend to upgrade to 
Internet within 12 months, which would reduce the total without Internet to about 1 1 per- 
cent of users. 

• Most NLM online users already have computer platforms that can support Internet appli- 
cations. Of the 81 percent of customers who use IBM-compatible computers, about 90 
percent have 386 or better processors; three-quarters have a 486 or Pentium machine. For 
the 1 8 percent who use Macintosh computers, about 90 percent are using System 7 soft- 
ware. 

• I lowever, a small but significant number of customers may still have serious equipment 
limitations. Even with planned customer upgrades, about 12 percent of users are pro- 
jected to still have 1.2 or 2.4 kilobaud modems, unacceptably slow for today's Internet 
applications. And 9 percent of IBM-compatible users are projected to still have a 286 or 
386 machine, again too limited for today's Internet. 



I yjj e surV cy results may actually understate the level of Internet access, since users of fixed-fee programs were ex- 
cluded from the survey, and Internet connectivity is a requirement of fixed-fee programs. 

" Accessible at http: 'mm.nlm.nih.gov Users must register to obtain an access code and are charged nominal access 
fees depending on level of use. 

"' Known as HyperDoc and accessible at http://www.nlm.nih.gov. 
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. The survey results indicate a high level of Windows use. For IBM-compatible users, all 
but about 13 percent have some version of Windows operating system. With upgrades, 
the percentage without Windows could fall to about 4 percent in 12 months, suggesting 
the use of Grateful Med for Windows as an interim solution for IBM-compatible custom- 
ers still without Internet. 

• Internet access is positively correlated with higher-end computer platforms. The faster 
the processor and modem and the larger the memory, the greater the likelihood that users 
have Internet access. 

• Over four-fifths of NLM customers are health care providers, librarians, and scientists: 
health care providers account for 46 percent of users; librarians 20 percent; and scientists 
1 9 percent. Educators totaled about 4 percent of users, and patients, consumers, students, 
and others about 1 1 percent collectively. 

• The office is the primary search location for about 47 percent of respondents, and the 
home is the primary location for 32 percent. For primary search location, health care 
providers search 49 percent from home and 40 percent from the office, while scientists 
search 79 percent from the office and 18 percent from home. Librarians search 56 per- 
cent from a library, 25 percent from the office, 12 percent from a hospital, and 7 percent 
from home. 

• Librarians are the most intensive users of NLM databases, with about three-fifths 
searching ten or more times a month 

• Overall, NLM customers indicated a high level of satisfaction with NLM online services. 
Sixty percent were very satisfied, and 34 percent moderately satisfied. Only 3 percent 
were moderately dissatisfied, and one percent very dissatisfied. 

The survey was mailed to a sample of 2,500 online users randomly selected from domes- 
tic billed users who searched NLM databases during the second quarter of 1995. The sampled 
population excluded users who could not be individually identified, such as users of fixed-fee 
programs and of NLM' s reading room. The final response rate was 83 percent of eligible re- 
spondents. 

NLM conducted the customer survey as part of its ongoing systems reinvention efforts. 
The survey was intended to: guide NLM's transition to Internet-based service delivery and spe- 
cifically Internet Grateful Med; identify user groups that may require support of older versions of 
Grateful Med for a period of time; and plan special outreach initiatives that may be needed to 
assure a more rapid and equitable transition from the old to the new forms of service delivery. 
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Introduction 

In February 1995, the National Library of Medicine (NLM) undertook to design and implement a 
survey of a representative sample of current online users. The primary purpose was to assess the 
availability of end-user technology capable of supporting an effective transition from Grateful 
Med to new forms of electronic information access and retrieval. Specifically. NLM wished to 
characterize users in terms of their: 

• searching behaviors and level of satisfaction with NLM services; 

• equipment platforms (hardware and software); 

• use of telecommunications, particularly the Internet; 

• equipment and telecommunications upgrades anticipated in the coming twelve 
months. 

The design of the survey instrument and the selection and wording of survey questions reflected 
NLM management needs to 

• predict the ability of NLM users to readily adopt a significantly different client-server 
search system (i.e., Internet Grateful Med) having high-end technology characteris- 
tics, 

• quantify the relative proportion of users that would still require costly support of older 
versions of GM; and 

• plan special outreach initiatives that ma> he needed to accomplish a more rapid tran- 
sition from the old to the new. 

The Survey Process 

An NLM team prepared the initial draft survey instrument. After several revisions, the instru- 
ment was pilot tested with a random sample of thirty users in September 1995. The final version 
of the survey was fielded by Macro International Inc. to a random sample of 2,500 U.S. flat-rate 
and billed users who searched NLM databases during the second quarter of 1995. The popula- 
tion included only users who conducted enough search activity to generate a bill. Also, NLM's 
reading room and fixed-fee program users were excluded. 

NLM extracted two user files: ( 1 ) 1 ,3 14 records of flat-rate program users; and (2) 1 7,754 rec- 
ords of regular billed users. Macro removed duplicates resulting in a population of 15,372 users, 
and then randomly drew 2.500 users. Questionnaires were mailed out to these 2,500 randomly 
selected users. Appendix B contains the cover letter; Appendix C the survey instrument. 

The first mailing took place in October 1995. with subsequent mailings to non-respondents m 
November and December. Follow-up phone calls were placed in January 1996. with data collec- 
tion terminated at the end of January. 
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Out of the 2,500 users sampled, 2,361 users were determined to be eligible NLM users. Of these 
2.361 users, 1,955 completed a least one questionnaire for a response rate of 82.8%, as shown in 
Table 1 . This is an exceptionally high response rate for a mail-out survey. Users who regularly 
search from a second location were requested to complete a second copy of the survey for that 
location. Ten respondents did so; thus, the 1,955 respondents returned at total of 1,965 surveys. 
Appendix D provides an overview of survey responses. 



Table 1. Survey Response Rate 



Original Random Sample 


2,500 


Could not be contacted by mail or phone 


139 


Eligible respondents 


2,361 


Completed one or more surveys 


1,955 


Response rate as percent of original sample 


78.2% 


Response rate as percent of eligible respondents 


82.8% 



Testing for Non-Response Bias 

Data were available to make comparisons for purposes of determining non-response bias across 
NLM regions, across states, and across Census Bureau Metropolitan Statistical Area types 
(henceforward described as MSA Types 1 ). Comparisons of the mail-out and response distribu- 
tions for the NLM regions (see Appendix E) and states are shown in Tables 2 and 3, respectively. 
"Goodness-Of-Fit" chi-square values 2 were calculated. There were no statistically significant 
differences in response rates across states 3 , including the District of Columbia and Puerto Rico, 
across known MSA Types 4 , or across NLM regions 5 . The high response rate (83%) and the sta- 
tistical results indicate that non-response bias is unlikely. 



Table 2. NLM Regional Response Pattern 



NLM Region 


Percent of Mail Out 


Percent of Responses 


Greater Midwest 


16.7% 


16.9% 


Middle Atlantic 


16.4% 


15.9%) 


Midcontinental 


5.5% 


5.7% 


New England 


8.3% 


9.0% 


Pacific N W 


5.9% 


6.2% 


Pacific SW 


15.5% 


15.0% 


SE/Atlantic 


23.3% 


22.8% 


South Central 


8.5% 


8.5% 


Total 


100.0% 


100.0% 



1 Respondent's ZIP code is either in or outside an MSA. If ZIP code is in an MSA, there are the following MSA 

Types: "inside a city center," "in city but outside center," "suburban" and "In MSA but outside city/suburban " 

" Refer to Appendix A 

3 Chi-square= 1 2.26,df=5 1 ,p>0.99 

4 Chi-square=5.71, df=4, p>0.22 

5 Chi-square=2.46, df=7, p>0.92 
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Table 3. State by State Response Pattern 





Mailed 


Response 




Mailed 


Response 




Mailed 


Response 




oul 










out 












out 










/o 


U 


0/ 

/o 




u 


% 


# 


0/ 

/o 




U 


% 


# 


0/ 

/o 


AK 


l 0 


\JM /o 


8 0.4% 




KY 


24 


1 .0% 


19 


1 .0% 




OH 


66 


2.6% 


47 


2.4° o 


AL 


~\A 


1 AO/ 
I A /o 


26 


1 .J /o 




LA 


47 


1 .9% 


42 


2.1% 




OK 


21 


0.8% 


18 


0.9% 


AR 


0 

o 


U.J /o 


5 


W.J /0 




MA 


94 


3.8% 


80 


4.1% 




OR 


28 


1.1% 


25 


1 .3% 


AZ 


A1 


1 "70/ 
1 . I/O 


37 


1 .9% 




MD 


148 


5.9% 


1 1 1 5.6% 




PA 


103 


4.1% 


79 


4.0% 


CA 


J 1 0 


1 0 TO/ 

1 Z. /vo 


237 


12.1% 




ME 


15 


0.6% 


14 


0.7% 




PR 


6 


0.2% 


5 


0.3% 


CO 


JJ 


1 10/ 

1 . Jyo 


29 


1.5% 




MI 


74 


3.0% 


62 


3.2% 




RI 


1 i 


0.4% 


9 


0.5% 


CT 


A7 


') <o/ 
Z.J/o 


54 


2.7% 




MN 


38 


1 .5% 


29 


1.5% 




SC 


16 


0.6% 


15 


0.8% 


DC 


zlQ 


7 AO/ 

Z.Uvo 


36 


1.8% 




MO 


45 


1 .8% 


38 


1 .9% 




SD 


4 


0.2% 


4 


0.2% 


DE 


1 1 
1 J 


U.J /o 


10 


0.5% 




MS 


18 


0.7% 


13 


0.7% 




TN 


38 


1.5% 


27 


1.4% 


FL 


88 
OO 


7 <s0/ 


69 


3.5% 




MT 


24 


1 .0% 


21 


1.1% 




TX 


1 10 


4.4% 


82 


4.2% 


GA 


53 


2.1% 


35 


1.8% 




NC 


A1 


1 70 

1 . / /o 


38 


1 .9% 




UT 


13 


0.5% 


11 0.6% 


HI 


16 


0.6% 


12 


0.6% 




ND 


12 


0.5% 


9 


0.5% 




VA 


76 


3.0% 


60 


3.1% 


IA 


28 


1.1% 


26 


1 .3% 




NE 


18 


0.7% 


15 


0.8% 




VT 


12 


0.5% 


1 1 0.6% 


ID 


15 


0.6% 


1 1 0.6% 




NH 


12 


0.5% 


9 


0.5% 




WA 


70 


2.8% 


56 


2.8% 


IL 


93 


3.7% 


72 


3.7% 




NJ 


95 


3.8% 


72 


3.7% 




WI 


46 


1.8% 


42 


2.1% 


IN 


31 


1 .2% 


22 


1.1% 




NM 


27 


1.1% 


20 


1 .0% 




WV 


14 


0.6% 


14 


0.7% 


KS 


24 


1 .0% 


17 


0.9% 




NV 


1 1 


0.4% 


8 0.4% 




WY 


4 


0.2% 


2 


0.1% 








NY 


198 


7.9% 


152 


7.7% 
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The NLM Customer Base 



NLM Online Users Profile 

l lealth care providers, librarians 
(including other information services 
professionals), and scientists constitute 
the largest segment of the NLM 
database users responding to the survey 
(see Figure 1 ). These three groups 
account for 85 percent of all 
respondents. Educators, media 
(including journalists), students, 
members of the legal profession, and 
health care consumers (e.g., patients) 
make up the remaining 1 5 percent. 



Who Are the NLM Online Users? 



Health Care Provider 
Librarian 
Scientist 
Other/Missing 
Educator 



Media 

HealthCare | 2% 
Consumer 1 

Student ■ 

I 1% 

Legal " 




46% 



40% 50% 



Figure 1. NLM Online Database Users 



User Group and 
Frequency of Use 



Total 



□ More than 10 times each month 

□ 4-10 times each month 
■ 1-3 times each month 

□ Less than once a month 



As a group, 19 percent of users search NLM databases less than once a month; 39 percent search 

one to three times each month; 23 
percent search four to ten times each 
month; and 1 9 percent search more 
than ten times each month (see 
Figure 2). By far the one group of 
users who search most frequently 
are librarians and other information 
services professionals, 58 percent of 
whom report searching more than 
ten times each month. Scientists as 
a group are the second most 
frequent searchers of NLM 
databases, of whom 45 percent 
report searching four or more times 
each month. This compares to 
librarians (and other information 
services professionals), 77 percent 
of whom report searching four or 
more times each month. Health 
care providers generally do less 
frequent searches than the other 
groups. 



Media 



Legal 



Librarian 



Student 
Educator 

Scientist 

Health Care 
Provider 




80% 90% 100% 



Figure 2. Users and Their Frequency of Use 
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Where Do NLM Users Do Their Searching? 



Search Locations: 
First and Second Most Frequent 




47% 



! 46% 



0% 

Other n 2% 



■ First 
□ Second 



o% 



10% 



20% 30% 40% 50% 



The office and the home are the 
most frequent locations for 
conducting NLM database 
searches (see Figure 3). The 
office is the primary search 
location for about 47 percent of 
respondents, and the home is the 
primary location for 32 percent. 
The pattern changes for the 
secondary search location, with 
46 percent of the users reporting 
searching the NLM databases 
from home as their second most 
frequent search location. 



Figure 3. First & Second Most Frequent Search Locations 



Figure 4 represents the 
Erequenc) of searching across 

the primary search locations. For example, looking at just the category search frequency "more 
than ten times each month," 47 percent occurs in libraries, and 35 percent occurs in offices. That 
is, in this category the most frequent searches are being done in libraries followed by offices. 
Considering the second most frequent category - tour to ten times each month, searching is being 
done from offices followed by homes. 



Most Frequent Search Location by Frequency of Searching 



70 0% 



60 0% 



50 0% 



40.0% 



30 0% 



20 0% 



10 0% 



0 0% 





Office Hospital Library Home Other 

Less than once a month ■ 1-3 times each month O4-10 times each a month □ More than 10 times a month 



Figure 4. Primary Search Location and Frequency of Searching 
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Figure 5 displays the ten 
largest user groups and 
primary search locations 
from among all 
respondents. Health care 
providers searching from 
the home and the office 
form the two largest 
groups, followed by 
scientists in their offices, 
librarians in libraries and 
offices, and then health 
care providers in 
hospitals. 

Figure 6 indicates the 
distribution of primary 
search locations across the 
user groups. This led to a 
separate analysis of just 
the health care providers, 
scientists, educators and 
librarians and their search 



Ten Largest User - Search Location Groups 










Health Care Provider - Home 






I 
















Health Care Provider - Office 


I 








- 










Scientist - Office 


























Librarian -Library 


I 








Librarian - Office 














I 










Hpalth Harp Providpr - 

1 ICO III V> QIC 1 1 UVIUCI 














Hospital 


I 
























Scientist - Home 














Other - Office 


























Librarian - Hospital 


I 












Educator - Office 














0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 
%of All Users 



Figure 5. The Ten Largest User-Search Location Groups 



90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% 



Most Frequent Search Location by User Groups 




Figure 6. Location of Search by User Groupings 
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locations. There are significant differences among these four user groups in both the primary 6 
and secondary search locations. Health care providers primarily search from home, whereas, 
not surprisingly, librarians primarily do their NLM database searching from libraries. Scientists 
search primarily from their offices. Health care providers search secondarily from their offices 
and hospitals, while scientists use libraries and homes for their second most frequent search 
location. 



Table 4. Most Frequent User-Search Locations 



Who are the most 
frequent users and 
their search loca- 
tions? The analysis 
provided in Table 4 
indicates that the 
librarian in a library 
is the largest single 
user-location 
grouping among the 
most frequent 
searchers of NLM 

databases (those searching ten or more times each month) . The numbers of respondents shown 
in Table 4 indicate the relative sizes of the various user-location-frequency of search groupings. 
Scientists searching from their offices four to ten times per month form the second largest 
grouping. 



4-10 Times Per Month 




10 or More Times Per Month 




Scientist - Office 




104 


Librarian - Library 




157 


Health Care Provider - 


Office 


87 


Librarian - Office 




43 


Health Care Provider - 


Home 


64 


Health Care Provider - 


Office 


35 


Librarian - Library 




32 


Scientist - Office 




33 


Librarian - Office 




26 


Librarian - Hospital 




25 


Health Care Provider - 


Hospital 


20 


Health Care Provider - 


Home 


13 


Educator - Office 




13 


Health Care Prov ider - 


Hospital 


9 


Other - Office 




12 


Scientist - Home 




8 


Scientist - Home 




1 1 


Other - Office 




7 


Librarian - Hospital 




1 1 


Health Care Provider - 


Library 


6 



Distribution of NLM Users 



MSA inside 
city center 
46% 



MSA: outside 
city center in 
city 
13% 




Geographic Distribution of Online 
Users 

As was described in the section on " The 
Survey Process," MSA Type is a derived 
variable based on the Census Bureau's 
metropolitan statistical areas and a 
respondent's ZIP code. Each respondent 
was asked to supply the ZIP code of the 
location where he or she does the most 
NLM online database searching. A match 
was made against a Genesys database that 
includes Census tracts and lists of ZIP codes that are associated with each tract. By this method 
most of the ZIP codes that were supplied are type cast as to the kind of census tract and whether 
the ZIP code is in a metropolitan statistical area (MSA). If the ZIP code is in the Genesys 
database, then the MSA Type can be determined. Figure 7 presents the distribution of NLM 
users across MSA Types. 



MSA outside 
Unknown Zip Quts|de MSA Clty/suburban 

5/0 12% 4% 



Figure 7. MSA Types of NLM Online Use rs 



6 Chi-square=977, df=9, p< 000I 

7 Chi-square-47, df=9, p< 000I 

" Table 4 is based on 1,965 responses from the 1,955 respondents. 
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Respondents were mostly from cities, including inside city centers (46 percent), and outside city 
centers ( 1 3 percent). Twelve percent were outside MSAs, and primarily in rural areas of the 
country. There are of course, rural areas inside many MSAs, and these rural areas are considered 
to be outside the city and suburban areas, but not outside the MSAs. This rural category included 
about four percent of the respondents. About five percent of the survey respondents could not be 
classified due to unknown ZIP codes. Geographical breakdowns are discussed in the section on 
the survey process. These include both states and the NLM regions (see Tables 2 and 3). 

Figure 8 indicates the primary search location for users in various MSA Types. In city centers 
the primary search location is the office. Hospital is used as a primary search location relatively 
more frequently in the areas outside of MSAs. 



Most Frequent Search Location by MSA Type 



60% 



50% 



40% 



30% 



20% 



10% 



0% 




City 
Center 



Outside 
Center 



Suburb 
in MSA 



In MSA 



Outside 
MSA 



Total 



I Office ■ Hospital □ Library ■Home □ Other 



Figure 8. Primary Search Location by MSA Type 



The largest user group - search location combination in each MSA Type is listed in Table 5. The 
health care provider is the largest user group in all Census Bureau defined MSA types (with 
known ZIP codes), with the search location being either office or home. The scientist searching 

from the office is the largest 
Table 5. User-Search Location across MSA Types group in the unknown ZI p 

code areas. Many universities 
have special ZIP codes that 
are not included in the Gene- 
sys database, which may ac- 
count for this anomaly. 



Census Bureau MSA Type 


User - Location of Use 


MSA: inside city center 


Health Care Provider - Office 


MSA: outside city center in city 


Health Care Provider - Home 


MSA: suburban 


Health Care Provider - Home 


MSA: outside city/suburban 


Health Care Provider - Office 


Outside MSA 


Health Care Provider - Home 


Unknown Zip 


Scientist - Office 
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The top 1 5 user-location-MSA- 
Type groupings were computed 
to better clarity the "who, what 
and where" of online users. 
Table 6 lists the 1 5 groupings 
in rank order by number of re- 
spondents. Health care provid- 
ers, scientists, and librarians in 
city centers make up the largest 
segments. 



Table 6. The Top 15 User-Location-MSA-Type Groupings 



Census MSA Type 


User 


Search 
Location 


Count 


City center 


Health Care Provider 


Office 


192 


City center 


Health Care Provider 


Home 


1 87 


City center 


Scientist 


Office 


145 


City center 


Librarian 


Library 


104 


Suburban 


Health Care Provider 


Home 


104 


Outside city center in MSA 


Health Care Provider 


Home 


67 


Suburban 


Health Care Provider 


Office 


60 


wUlolUC lVlo/-\ 


Health Care Provider 


Home 


JO 


Suburban 


Scientist 


Office 


50 


City center 


Librarian 


Office 


45 


Suburban 


Librarian 


Library 


39 


Outside MSA 


Health Care Provider 


Office 


39 


Outside city center in MSA 


Librarian 


Library 


35 


Outside city center in MSA 


Health Care Provider 


Office 


32 


Outside MSA 


Librarian 


Library 


32 



Figure 9 compares frequency of searching by rural versus urban users. The rural versus urban 
comparison indicates little difference in the frequency of searching NLM online databases. 
Indeed, there is no statistical difference between urban and rural users with respect to frequency 
of use . 



Frequency of Search by Rural v Urban 

H Less than once ■ 1-3 times □ 4-10 times □ More than 10 times 

each month 



Total 



Rural: Outside MSA 



Urban: Inside MSA 




0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Figure 9. Urban v Rural Comparison of Frequency of Online Searching 



9 Chi-square equals 5. 16, df=3, p>0. 16 
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NLM Region and Frequency of Use 



I Less than once ■ 1-3 times □ 4-10 times □ More than 10 times 
each month 



Mid Atlantic 
Greater Midwest 
New England 
Midcontmental 
Total 
South Central 
Pacific SW 
Pacific NW 
SE/Atlantic 

















1 






















































■ 


























HIHII 














1 1 , . S i 1 1 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Figure 10 portrays the dis- 
tribution of frequency of use 
across the NLM defined re- 
gions. The regions are listed 
in order of the combined 
percentages of four or more 
searches each month. The 
Mid Atlantic, Greater Mid- 
west and New England re- 
gions top the list in this re- 
gard. However, there is no 
statistical difference across 
NLM regions with respect to 
frequency of use. 10 



Figure 10. Frequency of Use Across NLM Regions 



Figure 1 1 compares the NLM re- 
gions in terms MSA Type compari- 
sons of their urban versus rural 
make-up. The Pacific Southwest 
Region is the most urbanized of the 
NLM regions, based on respondent 
locations, while the Pacific North- 
west is the most rural. The Greater 
Midwest and the Midcontinental 
regions are also more rural than 
other regions. While statistically 
significant, 1 there is no region that 
is specifically and predominately 
"rural." 



Chi-square=21.3, df=21, p>0.43 
Chi-square=l 14, df=7, pO.0001 



NLM Region and 
Urban Make-up 



Urban □ Rural 




Pacific NW 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Figure 11. NLM Regions and Urban v Rural Make-up 
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Grateful Med and Command Language Access 



Users of NLM Command Language 
and Grateful Med 

m DK/NS ■ Using the NLM command language □ Using Grateful Med 



Librarian 
Total 
Scientist 
Educator 
Media 

Legal 

Health Care 
Provider 

Student 



30% 



22% 



16% 



16% 



14% 



10% 



76% 



81% 



82% 



At the time for the survey, users 
had tw o main methods for 
searching the NLM databases — 
Grateful Med and the NLM com- 
mand language. Figure 12 dis- 
plays the survey results on search 
method. By far the one group us- 
ing the command language the 
most includes librarians (and other 
information services profession- 
als), with about 69 percent of this 
group using the NLM command 
language to search the online da- 
tabases. The difference in search 
methods used between librarians 
and the other groups is statistically 
significant . 

Among all other user groups, 
Grateful Med is the predominant 
access method, being favored by 

Figure 12. Users of Command Language and Grateful Med 81 t0 96 P ercent of each of the 

other groups. Only about 7 per- 
cent of health care providers use the command language to search NLM online databases. 



83% 



90% 



s 



92% 



96% 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Figure 1 3 describes the distri- 
bution of use of Grateful Med 
versus command language 
across the primary search lo- 
cations. Not surprisingly, us- 
ers in libraries are the heaviest 
users of the command lan- 
guage for NLM online data- 
base searching. Users 
searching from home use 
command language the least. 



Most Frequent Search Location 

I DK/NS ■Command language □ Grateful Med 



Home 
Hospital 
Library 
Office 
Other 



Y 




J 9% 


90% 






21% 


75% 




72% 






18% 


80% 






11% 


89% 


i , , , ; 1 1 1 1 1 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Figure 13. Primary Search Location and Access Mode 



Chi-square=629. df=16, p<0001 
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Figure 1 4 highlights a significant difference 13 in the methods used to search NLM databases 
between high frequency users (usually librarians and other information services professionals) 
and lower frequency users. High frequency users tend to use the command language to access 
the NLM databases. Lower frequency users rely heavily on Grateful Med to access NLM data- 

bases. 




Chi-square=355, df=6, p< 0001 
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IniernetAccess and Use: Currfnt and Prqjectfd 



Internet Access 



I dont have access 
to the Internet 

Ethernet, Novell or 
other LAN 

Dial-up with a 
modem 

Prodigy.AOL, 
CompuServe 

Dont know or not 
sure 



24.9% 



Levels of Internet Access 

Respondents were asked, 
"What type of access to the 
Internet do you have through 
the computer you most often 
use to search NLM's 
databases?" About 25 percent 
indicated that they believe 
they do not have Internet 
access. About four percent do 
not know or are not sure 
whether they have Internet 
access. The 1 ,93 1 respondents 
who answered the question 
provided 2, 1 90 responses. Fi S ure 1 5 - Modcs of Internet Access 

This indicates that many users have several modes of Internet access on one computer. As 
shown in Figure 15, the largest single mode of access is via the commercial services such as 
America Online, Prodigy, and CompuServe, etc. 




35.3% 



0 0% 5 0% 10 0% 15 0% 20 0% 25 0% 30 0% 35 0% 40 0 
Multiple Response: Percent of All Users with Mode of Access 



Figure 16. Internet Access and MSA Type 



City center 

Outside city 
center in city 

Suburban 

Outside 
city/suburban 

Outside Metro 
Statistical Area 

Unknown 
Geography 





□ Prodigy, AOL, CompuServe, etc 

□ Dial-up with a modem 

□ Ethernet, Novell or other LAN 

■ I dont have access to the Internet 

■ Dont know or not sure 



0.0% 2.0% 4 0% 6 0% 8.0% 10 0% 12 0% 14 0% 16 0% 



Figure 1 6 presents in 
overall percentages how 
the distribution of 
Internet access 
modalities varies across 
MSA Types. Since the 
largest number of 
respondents are from city 
centers, the largest 
number of all types of 
Internet access are in city 
centers as well. The 
largest number of users 
with no Internet access is 
also from city centers. 
But, as was confirmed 
with further analysis, 
those users outside 
MSAs have relatively 
less Internet access. 
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Figure 1 7 shows distribution of Internet access within each of the MSA Types. Figure 17 
amplifies the difference seen in Figure 16, showing that the respondents outside of MSAs have 
disproportionately less Internet access. 

Figure 17. Internet Access by MSA Type 



Internet Access by MSA Type 



45% 




Total City Outside Suburb In MSA Outside 

Center Center in MSA MSA 

■ No Access □ Ethernet, etc. □ Modem M Prodigy, etc. 



Figures 18 through 21 summarize the distribution of respondents with no Internet access across 
the key groups of variables listed. Figures 1 8 and 19 present the distribution of those 
respondents currently with no Internet access; Figures 20 and 21 present the projected 
distribution if respondents make their planned upgrades in Internet access. The Figures 18 
through 2 1 represent intervals of estimation called 95 percent confidence intervals. It is 
estimated that there is a 95 percent probability that the true percentage lies within such intervals. 
When such intervals do not overlap in ranges, the groups they represent are said to be 
significantly different in value on that variable. The key variables examined in Figure 18 
include: User group, first search location, MSA Type, urban v rural, NLM region of the country, 
frequency of search, computer support personnel, and computer type (IBM v Mac). One can 
readily see in Figure 18 that presently: 14 

• scientists as a group are more likely to have Internet access than librarians or health care 
providers, but not significantly different than educators; 

• those who use hospitals as primary search locations are less likely to have Internet access; 

• users who search from outside an MSA or in rural areas are less likely to have Internet 

access. 



14 A statistical summary, which provides observed significance levels, is provided in a Table 7. 
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95% Confidence Interval 
% No Internet Now 




0.0% 



10 0% 



20 0% 



30 0% 



40 OS 



50 0% 



60 0% 



Figure 18. Percentage of Respondents with No Internet Aecess Now by Ke> Variables 
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NLM researchers sought to determine if the significant difference in access between urban and 
rural areas was specific to particular NLM regions. A logistic regression analysis was carried 
out for this purpose. The results indicated that there was no unique contribution to predicting 
Internet access due to particular urban versus rural distinctions within the NLM regions. The 
only statistically significant geographical predictor of Internet access is the urban v rural 
distinction, which is independent of where in the country the respondent resides. 

Figure 19 is a continuation of Figure 18, and specifically looks at the relationship of computer 
capabilities and Internet access. The confidence intervals indicate that: 

• IBM-compatible computer users are less likely than Mac users to have Internet access; 

• users with more up-to-date equipment and operating systems are more like to have Internet 

access. 



Figure 19. Continued: Percentage of Respondents with No Internet Access Now by Key Variables 



COMPUTER TYPE 
Mac User 
IBM User 
IBM MEMORY 
More than 8 MB 
4 to 8 MB 
Under 4 megabyte 
PROCESSOR 
Pentium 
486 
386 

less than 386 
OPERATING SYSTEM 
Windows 95 
Windows 
Dos. with Window 
Dos only 
MODEM SPEED (ALL) 
28800 baud 
14400 baud 
9600 baud 
2400 baud 
1200 baud 



0.0% 



95% Confidence Interval 
% No Internet Now 




10.0% 



20.0% 



30.0% 



40.0% 



50.0% 



60.0% 



70.0% 



80.0% 



Refer to Appendix A 
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USER GROUP 
Health Care Providers 
Librarian 
Educator 
Scientist 

FIRST SEARCH LOCATION 
Library 
Home 
Office 
Hospital 
MSA TYPE 
Outside Standard Metro Area 
Outside city/suburban 
Suburban 
Outside city center 
Inside city center 
URBAN V RURAL 
Urban 
Rural 
NLM REGION 
Pacific NW 
Mid Atlantic 
Midcontlnental 
New England 
SE/Atlantic 
Greater Midwest 
Pacific SW 
South Central 
FREQUENCY OF SEARCH 
More than 10 limes a month 
4-10 times each month 
1-3 times each month 
Less than once a month 
PC SUPPORT PERSONNEL 
Support Avail 
Support Not Avail 
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95% Confidence 
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Projected % No 
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5.0% 
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15 0% 



20 0% 



25 0% 



30 0% 



35 0% 



Figure 20. Perc entage of Respondents with Projected No Internet Access by Key Variables 
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Figures 20 and 21 provide an analysis similar to Figures 18 and 19 except for the projected rather 
than current percentages of respondents without Internet access. Because of the uncertainty of 
respondent estimates for the future, projections for Internet access, while improving considera- 
bly, "muddle'' the differences in Internet access that exist now. However, it does appear that 
hospitals will continue to have limited Internet access, and significantly less access than will be 
a\ ailable from the home and office. Figure 21 indicates that Internet access will still be dispro- 
portionately higher for the users with the most up-to-date hardware and operating systems. The 
following users will have significantly 16 higher levels of Internet access: 

• IBM-compatible users with more than 4 MB of memory; 

• IBM-compatible users with access to Windows; 

• IBM-compatible users with 486 or Pentium processors; and 

• those with highest modem speeds (includes both IBM-compatible and Macintosh). 



95% Confidence Interval 
Projected % No Internet 




Figure 21. Continued: Percentage of Respondents with Projected No Internet Access by Key Variables 



Refer to Table 7 for the observed significance levels. 
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Table 7 provides the observed significance levels 17 from statistical tests relating current Internet 
access with other selected variables. For example, there is no significant difference across NLM 
regions with respect to the proportion of respondents who do not have Internet access now. It is 
important to NLM researchers to determine whether the relationships between key variables and 
Internet access is retained when the analysis is statistically controlled for the following four 
groups: hospital primary search locations, health care providers (HCPs), librarians, and rural re- 
spondents. 



Table 7. Observed Significance Levels for Internet Access by Selected Variables 







Controlling 


for: 








All 


Hospital 


HCPs 


Librarians 


Rural 


Currently No Internet by 


How do you most frequently search the NLM databases? 


18 

ns 


ns 


ns 


0.003 


ns 


IBM: How much memory... 


0.000 


ns 


0.000 


0.000 


0.048 


IBM: Type of processor... 


0.000 


ns 


0.000 


0.000 


ns 


IBM: Operating system of this computer... 


0.000 


ns 


0.000 


0.000 


0.027 


Mac: How much memory ... 


0.042 


ns 


ns 


ns 


ns 


Macintosh operating system... 


0.000 


ns 


0.001 


ns 


ns 


All: Highest speed of modem... 


0.000 


0.004 


0.000 


0.000 


0.001 


How often do you search the NLM databases? 


ns 


ns 


ns 


0.030 


ns 


Primary search location... 


0.000 


na 


0.000 


0.000 


0.025 


MSA Type... 


0.000 


ns 


0.014 


0.010 


na 


Urban versus Rural... 


0.000 


ns 


0.003 


0.013 


na 


NLM Region... 


ns 


ns 


ns 


ns 


na 


Computer support personnel available?... 


ns 


ns 


ns 


0.017 


ns 


Type of computer do you most often use... 


0.000 


ns 


0.006 


0.036 


ns 




Projected No Internet by 


How do you most frequently search the NLM databases? 


ns 


ns 


ns 


0.001 


0.038 


IBM: How much memory... 


0.000 


ns 


0.000 


0.001 


0.005 


IBM: Type of processor... 


0.000 


ns 


0.000 


0.000 


ns 


IBM: Operating system of this computer... 


0.000 


ns 


0.000 


0.001 


0.035 


Mac: How much memory ... 


ns 


ns 


0.042 


ns 


ns 


Macintosh operating system... 


ns 


ns 


ns 


ns 


ns 


All: Highest speed of modem... 


0.000 


0.002 


0.000 


0.000 


0.01 1 


How often do you search the NLM databases? 


ns 


ns 


ns 


0.033 


ns 


Primary search location... 


0.000 


na 


0.000 


ns 


ns 


MSA Type... 


ns 


ns 


ns 


ns 


na 


Urban versus Rural... 


ns 


ns 


ns 


ns 


na 


NLM Region... 


ns 


ns 


ns 


ns 


na 


Computer support personnel available?... 


ns 


ns 


ns 


ns 


ns 


Type of computer do you most often use... 


0.010 


ns 


ns 


ns 


ns 



17 Refer to Appendix A 

18 The note "ns" means "not significant." Refer to "statistical significance" in the glossary in Appendix A 

19 The note "na" "not applicable." Controlling for this variable reduces sample sizes too much or eliminates com- 
parisons. 
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1 he analvscs represented in Table 7 suggest several collusions. Compared to othe, • seareh 1 lo- 
cations, hospuais show consistently low Internet access independent of all key variables listed 
except modem speed. 

Among librarians (and other information services professionals) and health care providers, Inter- 
net access for the most part follows the same patterns of relationships as for all users. An excep- 
tion is among librarians and how they search the NLM databases. Those who use the command 
language are marginally less likely to have Internet access, and are less likely to have it in 12 
months, than those who use Grateful Med. 



Controlling for rural areas, the analysis of Table 7 
indicates that the low Internet access in rural areas 
is independent of all the listed variables except 
modem speed, where faster modem speed means 
more Internet access. Table 8 emphasizes the 
finding in Figure 18 concerning low Internet 
access in rural areas. The distinction is true in all 
search locations and most apparent in library 
search locations. 



Table 8. Rural-Search Locations and Percent 
Who Lack Internet Access 



Primary Search 
Location 


All MSA 

Types 


Rural 


Library 


26% 


49% 


Home 


20% 


30% 


Office 


24% 


29% 


Hospital 


48% 


51% 


All 


25% 


36% 



The distribution of Internet access at primary search locations will be changing in the next twelve 
months as users upgrade and different locations grow differentially in Internet access. Figure 22 
shows that home users will be advantaged with respect to Internet access, while hospitals will be 
relatively disadvantaged. 



Online Search Locations with/without Internet 
Access in 12 Months 



With 
Internet 

in 12 
months 



100% r 












90% 


34.2% 




23.1% 






80% 

70% - 








□ Home 

□ Hospital 






14.2% 




60% - 


5.4% 








50% 
40% 

30% } 










! □ Office 
■ Library 


20% 1 










\ ■ Other 


10% | 






0% 1 













Without 
Internet 
in 12 
months 



Figure 22. Distribution of NLM Searching by Internet Access 
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IBM Versus Macintosh Users and Internet Access 

The significant difference between IBM- 
compatible and Macintosh computer users 
with respect to Internet access (Figures 19 
and 21 and Table 7) warrants further analy- 
sis. If future NLM upgrades are oriented to- 
ward greater reliance on computer platforms 
running Windows or the equivalent with In- 
ternet access, then it is important to learn 
what will be required of present users to 
make that transition. Of the IBM- 
compatible users (N= 1,568 for this ques- 
tion), 1 9 percent have neither Windows nor 

Figure 23. IBM Users: Windows and Internet Access l"*™* ^ 1 3 P 6rCent haVe Win " 

dows available, but no Internet (see Figure 
23). Sixty-eight percent have both Windows and Internet access. This is in marked contrast to 
the Macintosh users (N=353), 80 percent of whom presently have Internet access (see Figure 24). 

Several key variables were examined to determine their impact on Internet access. For IBM- 
compatible users, there is no significant relationship 20 between Windows/Internet Access and 

• type of user, 

• frequency of use, or 

• availability of support personnel. 

For Macintosh users there is no relationship 21 
between Internet access and 

• type of user, or 

• frequency of use. 

22 

Macintosh users are more likely to have 
Internet access if computer support personnel 
are available. 

Figure 24. Macintosh Users and Internet Access 



Availability of Windows and 

Windows, 



Internet Access No 



Have 
Internet 
68% 




Macintosh Users & Internet 
Access 



No 
Internet 




~ All observed significance levels >0.05 

21 All observed significance levels >0.05 

22 Observed significance level <0.01 



National Library of Medicine 



Survey of Online Customers: Usage Patterns and Internet Readiness 



28 



Figure 25 summarizes much of the previous Internet discussion in a flow chart format. The chart 
indicates the percentages of those who do not presently have Internet access but intend to 
upgrade in the next twelve months. Also, there is a profile of those who have Internet access and 
use it to search NLM databases, those who have Internet access but do not use it, and those users 
\\ ho do not have Internet access. Refer to Table 7 for the observed significance levels for the 
Internet and no-Internet comparisons across these variables. 



Internet Access 
and Use 



Profile 

Health Care Providers 

Scientists 

Educators 

Librarians 

Students 

Others 

Urban 

Rural 

IBM-compatibles 
Pentium 
486 
386 
<386 
<4 MB 
4-8 MB 
>8 MB 
Windows 
DOS only 
All modems: 

28,800 baud 
14,400 baud 
9,600 baud 
2,400 baud 
1.200 baud 



Access to Internet 
75% 



No Access to Internet 
25% 



With Internet 
Access 

45% 
21% 

5% 
19% 

2% 

8% 
89% 
11% 

22% 
59% 
14% 
2% 
5% 
49% 
41% 
55% 
6% 

21% 
48% 
13% 
13% 
1% 



Use Internet to Access NLM 
Databases? 



Yes 
26% 

(20% of all 
respondents) 



User 

31% 
35% 
3% 
23% 
1% 
7% 
90% 
10% 

21% 
61% 
14% 

2% 

6% 
46% 
45% 
61% 

6% 

24% 
44% 
11% 
14% 
1% 



No 
74% 

(55% of all 
respondents) 



Non- 
user 

50% 

17% 

5% 

17% 

2% 

9% 

89% 

11% 

22% 

60% 

14% 

2% 

5% 

51% 

40% 

55% 

6% 

21% 
50% 
13% 
13% 
1% 



Upgrade to Internet within 12 
Months? 


Yes 




No 


58% 




42% 


(15% of 




(11% of 


respondents) 




respondents) 



Total with No Internet 
Access in 12 Months=11% 



No Internet Access 

49% 
12% 
4% 
23% 
1% 
11% 
82% 
18% 

9% 
48% 
26% 
11% 
15% 
47% 
22% 
42% 
25% 

6% 
25% 
21% 
31% 
7% 



Figure 25. Flow Chart of Internet Access and Use 



National Library of Medicine 



Survey of Online Customers: Usage Patterns and Internet Readiness 



29 



NLM users were asked what kind of Internet access they have now and what kinds of Internet 
access upgrades, if any, they were expecting in the next twelve months. About 25 percent of 
respondents do not have Internet access now. As can be seen in the Table 9, about eleven 
percent of respondents indicate that they do not have Internet access now and that they expect no 
upgrade. A more pessimistic estimate results from adding the "don't know/not sure" respondents 
to the "no upgrade expected respondents" for a total of about 19 percent 23 , who may not ha\ e 
Internet access in twelve months. This is a worst case estimate that assumes that all respondents 
who do not know or are not sure if they have Internet now or expect to upgrade in twelve months 
will not have Internet. Estimating within a 95 percent confidence interval, at best 8.5 to 12.9 
percent, and at worst 17 to 21.4 percent of NLM users will not have Internet access in twelve 
months. 

Table 9. Internet Access Now and in 12 Months 



In 12 Months 



Presently 


No Upgrade 
expected 


Ethernet, 
Novell, etc. 


Dial-up with a 
modem 


Prodigy, AOL, 
CompuServe 


Don't know 
or not sure 


No access to Internet 


10.7% 


2.0% 


2.4% 


4.1% 


6.2% 


Ethernet, Novell, etc. 


14.1%o 


2.5% 


0.5% 


0.4% 


4.6% 


Dial-up with a modem 


18.1% 


3.9% 


1 .6% 


1.1% 


3.9% 


Prodigy, AOL, CompuServe, 
etc. 


21.6% 


2.4% 


2.3% 


3.0% 


7.4% 


Don't know or not sure 


1 .4% 


0.0% 


0.4% 


0.2% 


2.3% 



(Note: Totals in Table 9 add up to more than 100% due to multiple responses) 

Respondents, who do not use the Internet to access NLM's online databases, were given the 
opportunity to specify why. For the 774 users who responded, Table 10 lists the reasons stated 
for not using the Internet. 



Table 10. Reasons for Not Using Internet 



Reason 


Percent of Respondents 




Who Indicated this Reason 


Prefer Grateful Med /direct access 


49% 


Can't figure out Internet 


17% 


Other reason 


9% 


Did not know one could 


7% 


Cost 


4% 


Just have not tried it 


4% 


Just got/will get Internet soon 


3% 


Modem too slow 


3% 


Reliability 


2% 


Don't know 


1% 


No need/No advantage 


1% 



23 Adding 10.7% + 6.2% + 2.3% = 19.2% 
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Access to The World Wide Web and Other Internet Features 

Of the respondents who indicated that they had access to the Internet via any type of local area 
network (LAN), a modem dial-up or a commercial gateway such as Prodigy, America Online, or 
CompuServe, questions were asked about what Internet services they had available and their ac- 
cess to the World Wide Web (WWW). Of the 1,339 respondents in this category, eight percent 
indicated that they either did not have access to WWW or they did not know. Table 1 1 indicates 
by multiple response frequencies the WWW access based on the type of Internet access the re- 
spondents have. Those who have access via a LAN or a modem dial-up predominantly use a 
Web browser, while those using a commercial gateway may be using whatever their provider has 
available. 



Table 1 1. WWW Access and Internet Access Mode 





Type of 


Internet Access 


Access to World Wide Web 


Local Area 
Network 


Modem 
Dial-up 


Commercial All 
Gateway 


No WWW access 


3% 


4% 


3% 3% 


Web browser 


86% 


75% 


41% 59% 


Dial-up with modem 


23% 


62% 


18%) 29% 


Prodigy, AOL, etc. 


21% 


35% 


87% 50% 


Don't know or not sure 


7% 


3% 


4% 5% 


Percents and totals based on 


1,339 respondents. Totals add to more than 100% due to multiple responses. 


Out of all the services provided via the Internet, e-mail is used the most across all access types. 
Table 1 2 indicates that well over 90 percent of those who access the Internet via LAN and mo- 
dem dial-up have used e-mail in the last year, and over 80 percent have used WWW. Those who 
access the Internet via the commercial gateways use e-mail and WWW proportionately less than 
the other two access modalities. 


Table 12. Internet Services Used and Internet Access 






Type of Internet Access 


Internet Service Actually 
Used in Last Year 


Local Area 
Network 


Modem Dial-up 


Commercial All 
Gateway 


E-mail 


91% 


94% 


77% 85% 


FTP 


57% 


53% 


29% 41% 


Telnet 


56% 


53% 


20% 38% 


Gopher 


65% 


63% 


44% 53% 


WWW 


85% 


84% 


73% 77% 


Other 


4% 


6% 


5% 5% 


Percents and totals based on 1,285 respondents. Totals add to more than 100% due to multiple responses. 
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Of those respondents who have Internet access, those searching NLM databases primarily from 
libraries and offices have used e-mail and WWW the most. Table 13 shows the distribution of 
Internet services used across the primary search locations. For example. 79 percent of those who 
search NLM databases primarily from their offices have used the WWW, and 87 percent have 
used Internet e-mail in the last year. 



Table 13. Internet Services Used and Search Location 



Primary Search Location 


Internet Service Actually 
Used in Last Year 


Office 


Hospital 


Library 


Home 


All 


E-mail 


87% 


81% 


90% 


80% 


85% 


FTP 


47% 


37% 


42% 


32% 


41% 


Telnet 


42% 


32% 


63% 


23% 


38% 


Gopher 


56% 


38% 


65% 


45% 


52% 


WWW 


79% 


68% 


80% 


74% 


77% 


Other 


5% 


5% 


4% 


5% 


5% 



Percents and totals based on 1,266 respondents. Totals add to more than 100% due to multiple responses. 



Of those who have Internet access, scientists and librarians have used e-mail and WWW the 
most. Table 14 shows the distribution across the major user groups. Among scientists, 81 per- 
cent have used the WWW, and 93 percent have used Internet e-mail in the last year; which is 
nearly the same proportions for librarians (and other information services professionals). 



Table 14. Internet Services Used and Major User Groups 



Major User Groups 


Internet Service Actually 


Health Care 


Scientists 


Librarians 


Others 


All 


Used in Last Year 


Providers 










E-mail 


79% 


93% 


92% 


80% 


85% 


FTP 


32% 


54% 


46% 


40% 


41% 


Telnet 


23% 


50% 


63% 


33% 


38% 


Gopher 


43% 


62% 


64% 


51% 


52% 


WWW 


75% 


81% 


80% 


74% 


77% 


Other 


6% 


3% 


5% 


6% 


5% 



Percents and totals based on 1,297 respondents. Totals add to more than 100% due to multiple responses. 

Of all the services listed, Telnet and FTP (File Transfer Protocol) are either the least used, or re- 
spondents are the least aware of having used them in the last year. Users may not be aware that 
their Web browsers may Telnet to a site, or download files via FTP. 
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End-User Computer Equipment: Current and Projected 



Users with IBM-Compatibles: Ready for Windows? 




Figure 26. Windows-Ready Computers 



Users with IBM-compatible computers were exam- 
ined as a group with regard to operating system (see 
Figure 26). It appears that all but about 1 3 percent 
are Windows capable. Twenty-seven percent have 
Windows available on their computers, although 
they use DOS. About 60 percent have Windows 
3.x or Windows 95 in use on their computers. 
Eleven percent have only DOS on their IBM- 
compatible computers. 

Figure 27 represents the distribution of IBM- 
compatible processors across MSA Types. The ru- 
ral group of users defined as those being outside an 
MSA appears low in the availability of Pentium 



computers and higher than average in 386s. The apparent differences, however, are not statisti- 
cally significant 24 . 



Distribution of IBM-compatibles by Metro Type 
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Figure 27. IBM-compatible Computers across MSA Types 



Chi-square=20.9, df=15, p>0.13 
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Current and Projected Distribution of 
IBM-compatible Processors 



□ less than 386 



386 D486 □ Pentium 



DK/NS 



Projected 



Current 




36% 




56% 



18% 
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Figure 28. IBM-compatible Processor Upgrades 



Current and Projected Distribution of 
IBM-compatible Operating System 

DOS ■Windows □ Windows 95 rjOS/2 □ DK/NS 



Projected 



Current 
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Figure 29. IBM-compatible Operating System Up- 
grades 



Current and Projected Distribution of 
IBM-compatible Memory 
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There is some uncertainty about upgrades in 
computer equipment over the next 12 months, 
since respondents are asked to predict future 
actions. The general picture that emerges is 
one of upgrading to higher-end equipment. 
As shown in Figure 28. after upgrades, less 
than ten percent of IBM-compatible proces- 
sors are expected to be a 386 or less. A de- 
crease in the number of 486 processors in use 
will be balanced out by a large increase in the 
number of Pentiums. 

Over the coming twelve months, IBM- 
compatible users expect to make significant 
upgrades to their operating systems. Figure 
29 combines the data on current operating 
systems with the twelve month projections. 
Those answering both questions will not be 
the same set of respondents answering just 
the present status question alone. The results 
suggest that there will be a migration away 
from DOS only usage to a greater reliance on 
Windows. About 32 percent of users indicate 
intentions to upgrade to Windows 95. About 
35 percent will be using Windows 3.x either 
by not upgrading or upgrading from DOS 
only usage. About 17 percent will be using 
DOS (but may have Windows available), 
which is down from the present 38 percent. 
About one percent will be using OS/2. 

With respect to computer memory, the results 
suggest that there is a strong intention to ob- 
tain more (see Figure 30). Over half of the 
IBM-compatible users suggest that they ex- 
pect to have eight megabytes or more of 
memory. This would go hand in hand with 
the large proportion of users intending to up- 
grade their processors and operating systems 
to Pentiums and Windows 95. 



Figure 30. IBM-compatible Memory Upgrades 
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Macintosh Equipment Upgrade Profiles 

Eighteen percent of computers used for online searching are Macintosh personal computers. 



Current and Projected Distribution of 
Macintosh Memory 



Current and Projected Distribution of 
Macintosh Operating System 
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More than 8 MB 
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System 7 
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Figure 31. Macintosh Memory and OS Upgrades 

Based on memory and operating systems described by respondents, there will not be the major 
amount of upgrades expected with the IBM and compatible personal computers. There will be 
some improvement in memory and some migration into the system 7 operating system. 



IBM v. Mac Comparisons 

Ninety-two percent of respondents 
use a modem to access NLM data- 
bases. With respect to modem 
speed, currently Macintosh users 
tend to have faster modems than 
IBM-compatible users, particularly 
the 28,800 baud speed. The pro- 
jected modem speeds are uncer- 
tain, but the trend appears to be 
away from 1 ,200 and 2,400 baud 
and towards the 28,800 baud mo- 
dems. This assumes that those 
who are unsure about upgrades 
will keep what they have now. 
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Figure 32. Current and Projected Modem Speed 

Note: Modem speed is in thousands of baud, or kilobits per second. 
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Table 15 describes the estimated percentage of users with various levels of computer platforms. 
Each category is read separately and does not indicate that the user's platform has all the charac- 
teristics listed. Of the IBM-compatible users, 52 percent have all the characteristics listed, while 
55 percent of the Macintosh users have all the characteristics listed. 



Table 15. User Computer Platforms 





IBM-compatible 

(81% of all respondents) 


Macintosh 

(18% of all respondents) 






%of 
IBM Users 




°oOf 

Mac Users 


Processor 


486/Pentium 


74% 






Memory 


4+ MB 


84% 


4+ MB 


93% 


Operating System 


Windows 


87% 


System 7 


90% 


Modem 


9,600 baud + 


73% 


9,600 baud + 


77% 



Each of these hardware and operating system elements add to the chances that the user will have 
access to the Internet. Figure 33 indicates that the proportion of IBM-compatible users with In- 
ternet access increases markedly as the equipment or operating systems is more up-to-date (refer 
to Table 7 for the observed significance levels). A logistic regression analysis was performed 
to test for confounding effects of these multiple features of the computer platforms. Could one 
variable be the determining factor, e.g. type of processor - 386, 486, Pentium? All these factors 
were included in the regression model, and it was found that each had a unique contribution to 
the chances of a user having Internet access. 
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IBM-compatible Platforms 
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< 386 
<4 MB 



DOS + Win 
9,600 Baud 
386 
4-8 MB 



Windows 
14.400 Baud 
486 
> 8 MB 



Windows 95 
28,800 Baud 
Pentium 



Figure 33. Internet Access and IBM-compatible Computer Platforms 



Refer to Appendix A 
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NLM Custq mfr Satisfaction 

Respondents were asked, "So far, are you satisfied with the online services NLM offers?" They 
were asked to rate their satisfaction on a four point scale ranging from "Very satisfied" to "Very 
dissatisfied." Respondents are 
for the most part satisfied 
with NLM services, since 91 
percent reported being either 
moderately or very satisfied. 
Four percent are dissatisfied 
to some degree, with only one 
percent very dissatisfied. 
This is a remarkably positive 
outcome for a customer satis- 
faction question. 

Since there are so few dissat- 
isfied customers, there are 
insufficient data to separate Figure 34. Satisfaction with NLM Online Services 
the four percent into a group 

to compare to those who are satisfied with the NLM services. Instead, satisfaction was treated as 
a continuous variable and statistically investigated to ascertain the factors associated with satis- 
faction. This was accomplished by using a one to four scale as follows: 

1 . Very satisfied 

2. Moderately satisfied 

3. Moderately dissatisfied 

4. Very dissatisfied 

A statistical technique known as analysis of variance (ANOVA - see Appendix A) was used to 
determine factors associated with satisfaction. A large number of independent variables were 
examined to determine the statistically significant factors associated with satisfaction. At NLM 
request, even variables that did not rise to a level of significant difference across membership in 
the categories of these variables were further analyzed. These are discussed in forms of charts 
and tables which indicate that levels of satisfaction are quite high regardless of the variable con- 
sidered. And in those few instances where statistically significant differences were found, the 
levels of satisfaction nevertheless were still quite high. 
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Figure 35 illustrates satisfaction with NLM online services as a function of NLM defined geo- 
graphical regions. The ANOVA indicates that region of the country was not a statistically sig- 
nificant 26 factor in satisfaction with NLM online services. The graph is represented using error 
bars with a 95% confidence interval. 27 The wider confidence intervals are all due to small sam- 
ple sizes of the particular region. 

Looking at the graph one can see that the confidence intervals for the most part "overlap" in their 
ranges of values. While Mid Atlantic and South Central regions differ the most in their apparent 
levels of satisfaction, the confidence intervals for the two regions share common values. The ap- 
parent difference is explained by the variation that results in random sampling from a population. 



Mid Atlantic 
Midcontinental 
New England 
Pacific SW 
Greater Midwest 
SE/Atlantic 
Pacific NW 
South Central 



Very 
Satisfied 




Satisfied 



Figure 35. Satisfaction across NLM Regions 



26 Refer to Appendix A 

27 F(7, 1808)= 1.44, p>0.18 
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Fiuure 36 indicates the non- 

• 28 

significant differences 
across user groups. Aver- 
age levels of satisfaction are 
nevertheless quite high 
across all groups and are 
essentially the same with in 
the precision of measure- 
ment. 



95% Confidence 

Interval on 
Satisfaction with 
NLM Online Services 




Other 



1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 

v Moderately 
Satisfied Satisfied 



Figure 36. Satisfaction by User Group 



As described previously, using the ZIP 
codes provided by respondents, it is 
possible to deduce the type of Census 
Bureau Metropolitan Statistical Area 
(MSA Type) the respondent is from. 
There are no statistically significant 
differences in satisfaction based on 
MSA Type. 
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Figure 37. Satisfaction across MSA Types 



28 F(9,1850)=0.90,p>0.52 

29 F(5, 1854)= 1.67, p>0.13 
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Rural: Outside 
MSA 

Urban: Inside 
MSA 
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Figure 38. Satisfaction across Urban v Rural types 



Refining the MSA 
Types into Rural ver- 
sus Urban (Figure 38) 
in order to detect dif- 
ferences in satisfaction 
still does not yield sta- 
tistically significant 
differences in satisfac- 
tion. 30 Here rural is 
defined as those areas 
outside the Census Bu- 
reau's MSAs. 



An ANOVA test was used to determine whether the respondent's state can predict satisfaction 
levels. Again there are no statistically significant differences. 31 Table 16 suggests an apparently 
large range of the state average satisfaction levels. However, the small state samples makes t or 
large errors of estimation that require even larger differences in satisfaction than exist here before 
a difference may be considered statistically significant. 



Table 16. States and Territories with Two or More Respondents & Satisfaction Rating 



State 


Avg. 
Satisfaction 


Count 




State 


Avg. 
Satisfaction 


Count 


State 


Avg. 
Satisfaction 


Count 


HI 


1.82 


1 1 




11. 


1.46 


69 


TN 


1.36 


25 


DE 


1.80 


10 




NJ 


1.46 


67 


NC 


1.35 


37 


ID 


1.80 


10 




WV 


1.46 


13 


IN 


1.33 


21 


UT 


1.70 


10 




AL 


1.46 


24 


MS 


1.33 


12 


ME 


1.69 


13 




VA 


1.45 


58 


NM 


1.33 


18 


NE 


1.69 


13 




MA 


1.45 


76 


KY 


1.31 


16 


VT 


1.67 


9 




AZ 


1.44 


36 


1A 


1.30 


23 


CO 


1.62 


29 




TX 


1.44 


77 


WA 


1.30 


54 


DC 


1.61 


33 




KS 


1.44 


16 


FL 


1.27 


64 


OR 


1.58 


24 




MI 


1.43 


60 


MT 


1.26 


19 


NH 


1.56 


9 




MO 


1.42 


33 


GA 


1.24 


33 


NY 


1.52 


145 




MD 


1.42 


105 


ND 


1 .22 


9 


OH 


1.51 


47 




CA 


1.42 


227 


PR 


1.20 


5 


SD 


1.50 


4 




CT 


1.41 


49 


OK 


1.17 


18 


MN 


1.48 


29 




AR 


1.40 


5 


NV 


1.13 


8 


LA 


1.48 


40 




SC 


1 40 


15 


Rl 


1.13 


8 


PA 


1.47 


75 




WI 


1.39 


38 


WY 


LOO 


2 






AK 


1.38 


8 





0 F( 1,1 765)= 1.35, p>0.05 
31 F(52, 1807)= 1.20, p>0.15 
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The following variables were examined as possible predictors of satisfaction, and all were 
not to be statistically significant: 



• 1 low often do you search the NLM 
databases? 

• Primary search location 

• How do you most frequently search 
the NLM databases? 

• Computer support personnel? 

• User group description 

• Type of computer most often used 

• Memory upgrade -IBM-compatible 

• Processor upgrade -IBM-compatible 

• Operating system upgrade - IBM- 
compatible 

• Amount of memory - Macintosh 



• Macintosh operating system 

• Memory upgrade -Macintosh 

• Operating system upgrade - Macin- 
tosh 

• Predicted upgrades 

• Use modem to search MEDLINE? 

• Highest speed of modem 

• Modem speed upgrade 

• Access to the Internet 

• Use Internet to access NLM data- 
bases? 

• Projected access to the Internet 



Several variables were found that predict variation in satisfaction. These variables involve the 
quality of IBM-compatible computer equipment that the users have available for online searches. 
In order to examine these relationships more closely a variable was created to describe low-end 
and high-end users: 

low-end users (n=33) 

DOS only, and 

less than 4 MB memory, and 

less than a 386 

high-end users (n=91) 

Windows 95 or OS/2, and 
more than 8 MB memory, and 
a 486 or Pentium 

A medium range user (n=l,356) is defined as one who is not in either of these two categories. 
Significant differences were found across these three groups - low, medium and high-end us- 
ers. 32 More specifically, differences in satisfaction were found based on the kind of IBM- 
compatible processor 33 and operating system 34 in the user's equipment platform. It should be 
emphasized that the levels of satisfaction among users with all types of equipment are quite high. 
However, the pattern is one of decreasing satisfaction with higher-end IBM-compatible users as 
shown in Figure 39. A similar pattern was not found for Macintosh users. This may reflect other 
factors not measured by this survey. None of these results point directly to a causal relationship 
determining satisfaction. For example, if the NLM databases are moved into configurations re- 



32 F(2,1477)= 5.69 p<004 

33 F(4, 1462)3.79 p<005 

34 F(5,1454)=4.10 p<.001 
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quiring more high-end environments, it is not clear this will "cause" a decline in overall satisfac- 
tion. Such a prediction would go beyond the data at hand. This may be a case where statistically 
significant differences have no practical significance, since the overall levels of satisfaction are 
so high. 
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Figure 39. Significant Differences in Satisfaction 
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Conclusions 

The survey process . The 83 percent final response rate as a percent of eligible respon- 
dents is exceptionally high for a mail-out survey. No statistically significant differences in re- 
sponse rates were found across states, Metropolitan Statistical Area (MSA) types, or NLM re- 
gions. The high overall response rate combined with these statistical results indicate that non- 
response bias is unlikely. 

NLM customer base . Health care providers, librarians (including other information 
services professionals), and scientists account, collectively, for 85 percent of NLM database us- 
ers responding to the survey. The majority of users search NLM databases three or fewer times a 
month. As a group, librarians and information services professionals search most frequently; 58 
percent report searching NLM databases more than ten times per month. Scientists as a group 
are the second most frequent searchers; 45 percent report searching four or more times per month 
(compared to 77 percent of librarians and information services professionals). 

The office and the home are the most frequent locations for conducting NLM database 
searches. The office is the primary search location for about 47 percent of respondents, and the 
home is the primary location for 32 percent. The pattern reverses for the secondary search loca- 
tion. The primary search locations for major user groups are: 

• health care providers-49 percent from home, 40 percent from the office; 8 percent 
from a hospital, and 3 percent from a library. 

• scientists— 79 percent from the office, 18 percent from home. 

• librarians-56 percent from a library, 25 percent from the office, 12 percent from a 
hospital, and 7 percent from home. 

The "librarian in a library" is the largest single user-location grouping among the most 
frequent searchers of NLM databases. Librarians also make far greater use of the NLM com- 
mand language (69 percent compared to 22 percent for all respondents), and less use of Grateful 
Med (30 percent compared to 76 percent for all respondents). 

About 12 percent of respondents conduct searches from outside MS As, and are thus de- 
fined as rural users. Survey results indicate no statistical difference between rural and urban us- 
ers with respect to frequency of use. Nor is there any statistical difference across NLM regions 
regarding frequency of use. 

Internet access and use. Overall, about 25 percent of respondents do not currently have 
Internet access and 75 percent do have access. Percentages of Internet access were compared for 
a wide range of variables. The results indicate that at present: 

• Scientists as a group are more likely to have Internet access than librarians or health 
care providers, but not significantly more likely than educators. 
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• Those who use hospitals as primary search locations are less likely to have Internet 
access. 

• Users who search from outside an MSA or in rural areas are less likely to have Inter- 
net access. 

• IBM-compatible computer users are less likely than Macintosh users to have Internet 
access. 



Users with the more up-to-date equipment and operating systems are more likely to 
have Internet access. 



Internet access does not appear to vary as a function of search frequency, NLM region, or 
availability of computer support personnel for IBM-compatible users. 

Overall, about 36 percent of rural respondents do not have Internet access, compared to 
25 percent of all respondents. Compared to other search locations, hospitals and rural libraries 
have the lowest Internet access, with 51 percent of rural hospitals (48 percent of all hospitals) and 
49 percent of rural libraries currently lacking access. 

Survey results indicate that, as users upgrade computer equipment over the next 12 
months, hospitals are likely to continue to be disadvantaged regarding Internet access. Internet 
access is likely to be disproportionally higher for users with the most up-to-date computer hard- 
ware and operating systems, including IBM-compatible users with access to Windows, 486 or 
Pentium processors, and more than 4 Megabytes of memory, and all users with the highest speed 
modems. 

Taken together, these results suggest the need for continuing special outreach efforts di- 
rected at rural and hospital-based users, rural libraries, and other users with limited Internet ac- 
cess or computer platforms. The results also reinforce the need for monitoring of user upgrades 
that would narrow gaps in Internet access. 

The majority (58 percent) of those without current Internet access indicated an intent to 
upgrade to Internet within 12 months. If these upgrades occur, the percentage of respondents 
without Internet access could decrease to as low as 1 1 percent. In the worst case, assuming all 
"don't know/not sure" respondents do not upgrade, the percentage without Internet access would 
be 19 percent. Estimating within a 95 percent confidence level, at best 8.5 to 12.9 percent and at 
worst 1 7 to 21 .4 percent of NLM online users will not have Internet access in twelve months. 

Most of the users with Internet are not currently using Internet to access NLM databases. 
Of the 75 percent of respondents with Internet, one-quarter are using Internet to access NLM da- 
tabases. However, about three-quarters are using Internet to access the World Wide Web and 85 
percent use Internet for electronic mail. This highlights the considerable potential for use of In- 
ternet to access NLM, as NLM databases become more visible and accessible via the Internet 
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Combined with the already high percentage of NLM users with Internet access, and the likeli- 
hood of further upgrades to Internet, the survey results provide a strong basis for NLM's transi- 
tion to Internet-based delivery options, such as Internet Grateful Med and NLM World Wide 
Web sues. 

Equipment profile . Overall, 81 percent of respondents use IBM-compatible personal 
computers, and 1 8 percent use Macintosh computers. 

For IBM-compatible users, all but about 13 percent have some version of the Windows 
operating system available on their computers, although another 27 percent have Windows but 
use DOS. With upgrades, in 12 months the percentage of IBM-compatible users without Win- 
dows could fall to about 4 percent. In other words, the Windows operating system will be almost 
universal among IBM-compatible users. This suggests the possibility of using Grateful Med for 
Windows as an interim solution for IBM-compatible customers still without Internet access. 

Projected upgrades in computer equipment over the next 12 months are inherently uncer- 
tain, given that respondents are asked to predict future actions. The general picture that emerges 
is one of upgrading to higher-end equipment. After upgrades, only about nine percent of IBM- 
compatible processors are expected to be 386 or less. The proportion of IBM-compatible com- 
puters with Pentium processors is projected to double to 36 percent. The proportion with more 
than eight megabytes of memory is expected to increase from 35 to 51 percent. And the propor- 
tion with modem speeds of 14.4 kilobaud or faster is projected to increase from 57 to 69 percent, 
with more than doubling of the 28.8 kilobaud modems (from 15 to 34 percent). 

Projected upgrades in Macintosh computers are less dramatic, since about 90 percent of 
Macintosh platforms already have a System 7 operating system and more than four megabytes of 
memory. The proportion with more than eight megabytes of memory is expected to increase 
from 57 to 69 percent, and with modem speeds of 14.4 kilobaud or faster from 67 to 76 percent 
(with a 50 percent increase in 28.8 kilobaud modems— from 28 to 45 percent). Macintosh users 
tend to have faster modems. 

NLM customer satisfaction . Respondents are for the most part satisfied with NLM online 
database services, since 91 percent reported being either moderately or very satisfied. Four per- 
cent are dissatisfied to some degree, with only one percent very dissatisfied. The levels of satis- 
faction do not vary significantly across user groups, locations, NLM regions, or levels of current 
or projected Internet access, among other variables. Users with high-end computer platforms 
(e.g., Windows 95, more than 8 megabytes of memory, 486 or Pentium processor) are somewhat 
less satisfied, but still have high levels of satisfaction (roughly midway between moderately and 
very satisfied). 
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Statistical Glossary 



This appendix provides a few brief definitions of standard statistical procedures. For a more de- 
tailed discussion of these concepts, the reader is referred to McClave, J.T., Dietrich II, F.H., Sta- 
tistics 3rd Ed. 1985, Dellen Publishing, Riverside NJ. 

Adjusted standardized residual 

The residual for a cell divided by an estimate of its standard deviation. 
Analysis of variance (ANOVA) 

A procedure that divides the total variation in the dependent variable into components (effects) and produces tests 
for the statistical significance of each component. This statistical technique allows the use of a dependent variable 
assumed to be measured with error, such as satisfaction in this case being limited to four values. The independent 
variables are typically categorical. There are two ways ANOVA may be accomplished, including so-called 
"parametric" and "non-parametric" techniques. The parametric techniques generally have greater statistical power, 
but the non-parametric ANOVA (Kruskal-Wallis H) can be used to verify any result. 

Chi-square (crosstabs, contingency table) 

Statistic used to test the hypothesis that the row and column variables are independent. Should not be used if any 
cell has an expected value less than 1, or if more than 20% of the cells have expected values less than 5. 

Confidence interval for a population parameter - 95% 

A range of values that 95% of the time includes the (true) population value of the parameter. 
Frequency 

The count or number of cases having each value (or range of values) of a variable. 
Goodness-Of-Fit 

A measure of how well the models fits the data. It is based on the squared differences between the observed and 
predicted probabilities. A small observed significance level for the goodness-of-fit statistic indicates that the model 
does not fit well. 

Kruskal-Wallis 1-way ANOVA 

Tests whether k independent samples are from the same population. Nonparametric equivalent of one-way 
ANOVA. 

Likelihood ratio chi-square 

A goodness of fit statistic similar to Pearson's chi-square. For large sample sizes, the two statistics are equivalent. 
The advantage of the likelihood-ratio chi-square is that it can be subdivided into interpretable parts that add up to 
the total. 

Logistic regression 

The probability of a dichotomous outcome variable is predicted from a set of independent variables The model is- 
Pr(event)=l/(l+exp(-(BO + B1*X1 + B2*X2 +...+ Bk*Xk))) 

Observed significance level 

The basis for deciding whether or not to reject the null hypothesis. It is the probability that a statistical result as 
extreme as the one observed would occur if the null hypothesis were true. If the observed significance level is 
small enough, usually less than 0.05 or 0.01, the null hypothesis is rejected. 



Nonparametric Statistical Tests 

Methods for comparing two or more populations that are based on an ordering of the sample measurements accord- 
ing to their relative magnitudes. These techniques require fewer or less stringent assumptions concerning the na- 
tures of the populations whence the samples were drawn than do the statistically more powerful parametric tests. 

Pearson chi-square 

The Pearson chi-square statistic is one of the ways to measure how well the selected model fits the data. It is the 
sum of squares of residuals, individually weighted by the reciprocal of their expected counts. When its significance 
level is small, it implies that the expected counts are too far from the observed counts, even adjusted for their size. 
For a saturated model, the Pearson chi-square statistic is zero. 

Standardized residual 

The standardized residual is the residual divided by the standard error of the observed count. Under the Poisson 
assumption, the sum of squares of standardized residuals is the Pearson chi-square statistic, hence standardized 
residuals are also known as Pearson residuals. They can therefore be used to check the individual contribution to 
the Pearson Chi-squares. Standardized residuals are asymptotically normal, with zero mean and a variance less 
than one. 

Statistical significance 

refers to apparent differences or relationships that are found that have small probabilities of chance occurrence. 
Typically, for single statistical test of probability, "small" has a rule of thumb value of 5 percent, but may be more 
or less depending on what is involved in the case of a wrong decision. In ex post facto research that five percent 
value is divided by the number of tests involved in order to reduce the number of spurious "findings. " For exam- 
ple, if 100 tests for significance are carried out, the '"small" probability becomes 0.05/100 or 0.0005. 

Standard error 

A measure of how much the value of a test statistic may vary from sample to sample. It is the standard deviation of 
the sampling distribution for a statistic. For example, the standard error of the mean is the standard deviation of the 
sample means. 

Wald (logistic regression) 

A statistic which tests the null hypothesis that a coefficient in a logistic regression model is zero. It has the undesir- 
able property that when the absolute value of the regression coefficient is large, the estimated standard error is too 
large and the Wald statistic is too small. Whenever you have a large coefficient, do not rely on the Wald statistic 
for hypothesis testing. Instead look at the improvement in chi-square when the variable is added to the model. 
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National Library of Medicine 






Donald A.B. Lindberg, M.D., Director 


'<£<&al 







October 30, 1995 



Dear Colleague: 

The National Library of Medicine (NLM) would appreciate your assistance by responding 
to a brief questionnaire about how you, our customer, currently access our information 
services. You have been selected as part of a nationally representative sample of our 
online users. 

While we are confident that you will be pleased with improvements we are planning, we 
need to get your answers to some questions in order to be certain that you can take 
advantage of the new developments. The survey is one aspect of a continuing effort to 
improve the quality and availability of NLM's online services based on customer 
feedback. This survey is authorized under the Government Performance and Results Act 
(PL 103-62). Your cooperation is entirely voluntary, but it is very important to us. 

The questionnaire will take about five to ten minutes to complete. Your answers will not 
be maintained with any personally identifying information and will be reported only in 
aggregate form. In addition to answering the questionnaire, we welcome any suggestions 
you have for improving NLM's products, services, and delivery systems. 

Please fill out and return your questionnaire within one week using the enclosed business 
reply envelope. If you need additional information, please call (800) 639-2030 between 
9:00 am and 9:00 pm eastern time. 

Thank you for your help. 




Donald A.B. Lindberg, M.D. 
Director 

Enclosures 



Appendix C 
Survey Questionnaire 



NATIONAL LIBRARY OF MEDICINE 



OMB No: 0937-0201 
Exp: 6/30/96 



CUSTOMER SURVEY 



As part of a continuing effort to improve the quality and availability of its online services, the National Library 
of Medicine (NLM) would appreciate your taking a few minutes to tell us about the information technologies 
you currently use and those you anticipate using in the future. As a government agency, we would also like 
for you to tell us how you, our customer, feel you are presently being served. 

For each question, circle the one best answer unless the instructions specifically ask you to circle all answers 
that apply. You may wish to confer with a technical person in your organization to answer some questions. 
If you have any questions about this questionnaire, you may call (800) 639-2030 between 9 a.m. and 9 p.m. 
eastern time. Thank you! 



1. First, have you used MEDLINE and/or 
other NLM databases at any time in the 
past 12 months? 

a. Yes 

b. No 



If you marked "No" to Question 1 , stop here 
and return your questionnaire in the enclosed 
business reply envelope. If you marked "yes", 
proceed to Question 2. 



2. How often do you search the NLM 
databases? 

a. Less than once a month 

b. 1-3 times each month 

c. 4-10 times each month 

d. More than 10 times each month 

3. From what location(s) do you conduct 
searches? Rank order by frequency. Use 
"1 " for the most frequent, "2" for the 
second most frequent, etc. 



5. Do you have computer support personnel 
and/or a network/system administrator to 
help with hardware and software 
installations? 

a. Yes 

b. No 

c. Don't know or not sure 

6. Please identify the one user group that best 
describes you: 



a. 


Practicing physician, nurse, or other 




health professional 


b. 


Scientist 


c. 


Educator 


d. 


Student 


e. 


Librarian or other information services 




professional 


f. 


Patient or other health care consumer 


9- 


Other (specify) 



7. Please enter the zip code of the location 
from which you do most of your searching. 



Office 8. 

Hospital 

Library 

Home 

Other (specify) 

4. How do you most frequently search the 
NLM databases? 

a. Using Grateful Med 

b. Using the NLM command language 

c. Don't know or not sure 



What one type of computer do you most 
often use to search MEDLINE and other 
NLM databases? 



a. 


IBM-compatible 


nr 


GO TO Q9 


b. 


Macintosh 


rs* 


GO TO Q1 5 


c. 


Unix 


*s* 


GO TO Q22 


d. 


Dumb terminal only; 








no computer 


•s* 


GO TO Q19 


e. 


Don't know or not sure 


rr 


GO TO Q22 



«• TURN PAGE OVER 



PubEc reporting burden for th« co0.ct.on of nformetwn is estimated to average 7 minute, per response, including, the time for reviewing instructions, searching 
existing data sources, gathering and maintaining the data needed, end completing and reviewing the collection of information. Send comments regarding this 
burden estimate or any other aspect of f Ita collection of information, including suggestions for reducing this burden, to DHHS Reports Clearance Officer; 
Paperwork Reduction Project (0937-0201); Room 531-H; Humphrey Building; 200 Independence Ave.. SW; Washington. DC 20201. An agency may not 
conduct or sponsor, and a person ,s not required to respond to. a collection of information unless it displays a currently valid OMB control number. The OMB 
control number for this project is 0937-0201 . 



IBM-COMPATIBLE PC USERS ONLY Q9-14 



9. 



10. 



11 



What type of processor does this 
computer have now? 

a. Less than a 386 processor 

b. 386 processor 

c. 486 processor 

d. Pentium 

e. Don't know or not sure 

How much memory does this computer 
have now? 

a. Under 4 megabytes (MB) 

b. 4 to 8 MB 

c. More than 8 MB 

d. Don't know or not sure 

What operating system do you use on this 
computer now? 



12. 



13. 



a. 

b. 
c. 



I use Windows, Windows for 

Workgroups, or Windows NT 

I use Windows '95 

I use DOS, but the computer can run 

Windows 

I use DOS only, and the computer 

doesn't have Windows 

OS/2 

Don't know or not sure 



14. 



Within the next 12 months, to what type of 
processor do you expect this computer or 
its replacement will be upgraded? 



a. 
b. 
c. 
d. 
e. 



Processor is not likely to be upgraded 
Will be upgraded to 386 processor 
Will be upgraded to 486 processor 
Will be upgraded to Pentium 
Don't know or not sure 



Within the next 12 months, after upgrades, 
how much memory do you expect this 
computer or its replacement will have? 

a. Memory will not be upgraded 

b. 4 to 8 MB 

c. More than 8 MB 

d. Don't know or not sure 

Within the next 12 months, to what type of 
operating system do you expect this 
computer or its replacement will be 
upgraded? 

a. Operating system will not be upgraded 

b. Windows, Windows for Workgroups, or 
Windows NT 

c. Windows '95 

d. OS/2 

e. Don't know or not sure 

«-GO TO Q21 



15. 



16. 



MACINTOSH USERS ONLY Q15-18 



How much memory does this Macintosh 17. 
computer have now? 

a. Under 4 megabytes (MB) 

b. 4 to 8 MB 

c. More than 8 MB 

d. Don't know or not sure 



What operating system does this 
computer have now? 

a. Less than System 7 

b. System 7 

c. Don't know or not sure 



18. 



Within the next 12 months, after upgrades, 
how much memory do you expect this 
computer or its replacement will have? 

a. Memory will not be upgraded 

b. 4 to 8 MB 

c. More than 8 MB 

d. Don't know or not sure 

Within the next 12 months, to what type of 
operating system do you expect this 
computer or its replacement will be 
upgraded? 

a. Operating system will not be upgraded 

b. System 7 

c. Don't know or not sure 

«rGO TO Q21 



19. 



DUMB TERMINAL USERS ONLY Q19-20 

What type of dumb terminal do you use? 20. 



a. Emulates VT-100 

b. Emulates VT-1 02 

c. Don't know or not sure 



Within the next 12 months, to what 
type of computer do you expect this 
dumb terminal will be upgraded? 



a. 
b. 
c. 
d. 
e. 



Will not be upgraded 
Macintosh 
IBM-compatible 
Other: 



Don't know or not sure 



ALL USERS CAN ANSWER Q21-35 26. 

21. How confident are you that your predicted 
upgrades in type of processor, memory, or 
operating systems (if any) will actually 
occur in the next 12 months? 

a. No upgrades expected 

b. Virtually certain 

c. Probable 

d. Possible 

22. Please circle any features your computer 
possesses. 

CIRCLE ALL THAT APPLY. 

a. Has a color monitor 

b. Is a lap top 27. 

23. Do you use a modem to search MEDLINE 
and other NLM databases? 



a. Yes 

b. No 

c. Don't know or not sure 



w GO TO Q26 
■r GO TO Q26 



What type of access to the Internet do 
you have through the computer you 
most often use to search NLM's 
databases? 

CIRCLE ALL THAT APPLY. 

a. I don't have Internet access 

«• GO TO Q31 

b. Ethernet, Novell or other local area 
network (LAN) 

c. Dial-up with a modem (e.g. using 
SUP or PPP) 

d. Prodigy, America Online, 
CompuServe, or other gateways 

e. Don't know or not sure 

«• GO TO Q31 

Do you currently use the Internet as a 
communications link to access NLM's 
online databases? 

a. Yes 

b. No Please specify why not: 



24. What is the highest speed this modem can 
use now for data (not FAX) connections? 

a. 1200 baud 

b. 2400 baud 

c. 9600 baud 

d. 14400 baud 

e. 28800 baud 

f. Don't know or not sure 

25. Within the next 12 months, to what speed 
do you expect this modem will be 
upgraded or replaced? 

a. No upgrades to modem speed are 
expected 

b. 2400 baud 

c. 9600 baud 

d. 14400 baud 

e. 28800 baud 

f. Don't know or not sure 



28. What services can you access on the 
Internet, whether or not you are using 
them now! 

CIRCLE ALL THAT APPLY. 

a. E-mail 

b. FTP 

c. Telnet 

d. Gopher 

e. World Wide Web 

f. Other: (please specify) 

g. Don't know or not sure 



Through what mechanisms do you have 
the capability to access the World Wide 
Web, even if you aren't using this 
computer capability now? 
CIRCLE ALL THAT APPLY. 

a. No capability to access the World Wide 
Web 

b. Web browser, such as Mosaic, 
Netscape, Web Explorer 

c. Dial-up with modem (e.g. using SLIP or 
PPP) 

d. Prodigy; America Online, CompuServe, 
or other gateways 

e. Don't know or not sure 

What services on the Internet have you 
actually used in the past 12 months at any 
time? CIRCLE ALL THAT APPLY. 

a. E-mail 

b. FTP 

c. Telnet 

d. Gopher 

e. World Wide Web 

f. Other: (please specify) 

g. Don't know or not sure 

Within the next 12 months, what upgrades 
in Internet access do you expect that the 
computer you use to search NLM's 
databases or its replacement will receive? 
CIRCLE ALL THAT APPLY. 

a. No upgrades in Internet access 
expected » GO TO Q33 

b. Ethernet, Novell or other local area 
network (LAN) 

c. Dial-up with a modem (e.g. using SLIP 
or PPP) 

d. Prodigy, America Online, CompuServe, 
or other gateways 

e. Don't know or not sure «* GO TO 033 



32. How confident are you that your 
predicted upgrades in Internet access 
will actually occur? 

a. Virtually certain 

b. Probable 

c. Possible 

33. So far, are you satisfied with the online 
services NLM offers? 

a. Very satisfied 

b. Moderately satisfied 

c. Moderately dissatisfied 

d. Very dissatisfied 

e. No opinion 

34. Please indicate the reasons you use 
NLM's online services. 

CIRCLE ALL THAT APPLY. 

a. Easy to use 

b. Cost-effective 

c. Convenient 

d. Quality of customer support 

e. Best option available 

f. Other (specify) 



NOTE: If you regularly search 
NLM's databases from a second 
location, please photocopy this 
questionnaire and respond to 
questions 7 through 32 for the 
second location using a different 
color ink. Please return both 
questionnaires together. 



Thank you for completing this survey. 
Please use the enclosed business reply envelope to return the survey to 
National Library of Medicine Customer Survey, 
126 College Street, Suite 2A, Burlington, VT 05401. 



Please attach any additional comments or suggestions 
you think would be helpful in NLM's continuous efforts 
to improve customer service. 



Appendix D 
urvey Response At A Glan 



\. l sera 



l scr groups 



Most frequent search location 



Search frequency 



Search method 



Health care provider 
Librarian 
Scientist 
Educator 

Patient /health care consumer 

Student 

Legal 

Media 

Other/Missing 

Office 

Hospital 

Library 

Home 

Other 

Less than once a month 

1-3 times each month 

4-10 times each month 

More than 10 times each month 

Grateful Med 

NLM command language 



Percent of all users responding 

46% 

20% 

19% 

4% 

2% 

1% 

1% 

2% 

5% 

47% 

8% 
13% 
32% 

2% 

19% 
39% 
23% 
19% 

77% 
22% 



B. Equipment Platform 

Type of computer used most often 



IBM-compatible 
Macintosh 



Percent of all users responding 

81% 

18% 



B-I. IBM Compatible Users 

Processor 



Memory 



Less than a 386 

386 

486 

Pentium 

Under 4 MB 
4 to 8 MB 
More than 8 MB 



Percent of IBM-compatible users 
responding 
5% 
17% 
56% 
18% 

8% 
49% 
35% 



B-l. IBM Compatible Users - continued 



Operating system 

DOS without Windows 11% 

Windows -total 87% 

DOS with Windows available 27% 

Windows 51% 

Windows 95 9% 



B-2 Macintosh Users 

Memory 

Operating System 



Under 4 MB 2% 
4 to 8 MB 36% 
More than 8 MB 57% 



Less than System 7 6% 
System 7 89% 



B-3 Modem Use 

Modem used to search 

Modem Speed 



Yes 
No 

1200 Baud 
2400 Baud 
9600 Baud 
14400 Baud 
28800 Baud 



Percent of all users responding 
92% 
7% 

3% 
18% 
15% 
42% 
17% 



( . Internet Vccess end Use 



Type of Internet Access 



Currently us Internet as commu- 
nications link with NLM 



Internet services available 



No access to the Internet 
Internet Access 

LAN 

Dial-up Modem 
Prodigy, AOL, CompuServe, etc. 



No 
Yes 

E-mail 
FTP 
Telnet 
Gopher 

World Wide Web 



Percent of all respondents 

25% 

75% 

22% 

28% 

35% 

Percent with Internet access 

74% 
26% 

91% 
62% 
60% 
73% 
84% 



Internet services used 



E-mail 
FTP 
Telnet 
Gopher 

World Wide Web 



85% 
41% 
38% 
52% 
77% 



D. User Satisfaction 



Satisfaction with online services 
NLM offers 



Very satisfied 
Moderately satisfied 
Moderately dissatisfied 
Very dissatisfied 
No opinion 



Percent of all users responding 

58% 
33% 

3% 

1% 

5% 



Appendix E 

Geographic Areas Served by the Regional Medical Libraries 



Following is a list of the Regional Medical Libraries and the areas served by each. 



1. Middle Atlantic Region 

The New York Academy of Medicine 

1216 Fifth Avenue 

New York, New York 10029 

Phone: 212-876-8763 

FAX: 212-534-7042 

email: RML1@nyam.org 

States served: Delaware, New Jersey, 

New York, Pennsylvania 

ONLINE CENTER for all Regions 

2. Southeastern/Atlantic Region 

University of Maryland at Baltimore 
Health Sciences Library 
1 1 1 South Greene Street 
Baltimore, Maryland 21201-1583 
Phone: 410-706-2855 
FAX: 410-706-0099 

States served: Alabama, Florida, Georgia, 
Maryland, Mississippi, North Carolina, South 
Carolina, Tennessee, Virginia, West Virginia, 
the District of Columbia, Puerto Rico, 
and the U.S. Virgin Islands 

3. Greater Midwest Region 

University of Illinois at Chicago 

Library of the Health Sciences (M/C 763) 

1750 West Polk Street 

Chicago, Illinois 60612-7223 

Phone: 312-996-2464 

FAX: 312-996-2226 

List-serv: GMR-lnfo@uic.edu 

States served: Iowa, Illinois, Indiana, Kentucky, 

Michigan, Minnesota, North Dakota, Ohio, 

South Dakota, and Wisconsin 

4. Midcontinental Region 

University of Nebraska Medical Center 
Leon S. McGoogan Library of Medicine 
600 South 42nd Street 
Omaha, Nebraska 68198-6706 
Phone: 402-559-4326 
FAX: 402-559-5482 

States served: Colorado, Kansas, Missouri, 
Nebraska, Utah, and Wyoming 



5. South Central Region 

Houston Academy of Medicine- 
Texas Medical Center Library 
1133 M.D. Anderson Boulevard 
Houston, Texas 77030 
Phone: 713-790-7053 
FAX: 713-790-7030 
email: #nnlm@library.tmc.edu 
States served: Arkansas, Louisiana, New 
Mexico, Oklahoma, and Texas 

6. Pacific Northwest Region 

Health Sciences Library 
University of Washington 
Box 357155 

Seattle, WA 98195-7155 

Phone: 206-543-8262 

FAX: 206-543-2469 

email: nnlm@u.washington.edu 

States served: Alaska, Idaho, Montana, 

Oregon, and Washington 



7. Pacific Southwest Region 

University of California at Los Angeles 

Louise Darling Biomedical Library 

10833 Le Conte Avenue 

Los Angeles, California 90024-1798 

Phone: 310-825-1200 

FAX: 310-825-5389 

States served: Arizona, California, Hawaii, 
Nevada, and U.S. Territories in the Pacific 
Basin 



8. New England Region 

University of Connecticut Health Center 
Lyman Maynard Stowe Library 
263 Farmington Avenue 
Farmington, Connecticut 06034-4003 
Phone: 203-679-4500 
FAX: 203-679-1305 
States served: Connecticut, Maine, 
Massachusetts, New Hampshire, 
Rhode Island, and Vermont 
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